Stochastic Gradient Descent in Continuous Time

Justin Sirignano* and Konstantinos Spiliopoulos $^{† ‡}$

October 31, 2017

Abstract

Stochastic gradient descent in continuous time (SGDCT) provides a computationally efficient method for the statistical learning of continuous-time models, which are widely used in science, engineering, and finance. The SGDCT algorithm follows a (noisy) descent direction along a continuous stream of data. SGDCT performs an online parameter update in continuous time, with the parameter updates $θ_{t}$ satisfying a stochastic differential equation. We prove that $lim_{t \to \infty} \nabla \bar{g} (θ_{t}) = 0$ where $\bar{g}$ is a natural objective function for the estimation of the continuous-time dynamics. The convergence proof leverages ergodicity by using an appropriate Poisson equation to help describe the evolution of the parameters for large times. For certain continuous-time problems, SGDCT has some promising advantages compared to a traditional stochastic gradient descent algorithm. This paper mainly focuses on applications in finance, such as model estimation for stocks, bonds, interest rates, and financial derivatives. SGDCT can also be used for the optimization of high-dimensional continuous-time models, such as American options. As an example application, SGDCT is combined with a deep neural network to price high-dimensional American options (up to 100 dimensions).

1. Introduction

This paper develops a statistical learning algorithm for continuous-time models, which are common in science, engineering, and finance. We study its theoretical convergence properties as well as its computational performance in a number of benchmark problems. Although the method is broadly applicable, this paper mainly focuses on applications in finance. Given a continuous stream of data, stochastic gradient descent in continuous time (SGDCT) can estimate unknown parameters or functions in stochastic differential equation (SDE) models for stocks, bonds, interest rates, and financial derivatives. The statistical learning algorithm can also be used for the optimization of high-dimensional continuous-time models, such as American options. High-dimensional American options have been a longstanding computational challenge in finance. SGDCT is able to accurately solve American options even in 100 dimensions.

Batch optimization for the statistical estimation of continuous-time models can be impractical for large datasets where observations occur over a long period of time. Batch optimization takes a sequence of descent steps for the model error for the entire observed data path. Since each descent step is for the model error for the entire observed data path, batch optimization is slow (sometimes impractically slow) for long periods of time or models which are computationally costly to evaluate (e.g., partial differential equations). Typical existing approaches in the financial statistics literature use batch optimization.

SGDCT provides a computationally efficient method for statistical learning over long time periods and for complex models. SGDCT continuously follows a (noisy) descent direction along the path of the observation; this results in much more rapid convergence. Parameters are updated online in continuous time, with the parameter updates

θ_{t}

satisfying a stochastic differential equation. We prove that

lim_{t \to \infty} \nabla \bar{g} (θ_{t}) = 0

where

\bar{g}

is a natural objective function for the estimation of the continuous-time dynamics.

Consider a diffusion

X_{t} \in X = R^{m}

d X_{t} = f^{*} (X_{t}) d t + σ d W_{t} .

The goal is to statistically estimate a model

f (x, θ)

for

f^{*} (x)

where

θ \in R^{n}

. The function

f^{*} (x)

is unknown.

W_{t} \in R^{m}

is a standard Brownian motion. The diffusion term

W_{t}

represents any random behavior of the system or environment. The functions

f (x, θ)

and

f^{*} (x)

may be non-convex.

The stochastic gradient descent update in continuous time follows the SDE:

d θ_{t} = α_{t} [\nabla_{θ} f (X_{t}, θ_{t}) {(σ σ^{⊤})}^{- 1} d X_{t} - \nabla_{θ} f (X_{t}, θ_{t}) {(σ σ^{⊤})}^{- 1} f (X_{t}, θ_{t}) d t],

where

\nabla_{θ} f (X_{t}; θ_{t})

is matrix valued and

α_{t}

is the learning rate. The parameter update 1.2 can be used for both statistical estimation given previously observed data as well as online learning (i.e., statistical estimation in real-time as data becomes available). SGDCT will still converge if

σ σ^{⊤}

in 1.2 is replaced by the identity matrix

I

Using the proposed approach of this paper, the stochastic gradient descent algorithm (1.2) can also be generalized to the case where

σ

is a variable coefficient

σ^{*} (X_{t})

. In that case, a model

σ (x, ν)

is also learned for

σ^{*} (x)

where

ν \in R^{k}

is an additional set of parameters. (To be more precise,

σ (x, ν) σ^{⊤} (x, ν)

is learned for

σ^{*} (x) σ^{*, ⊤} (x)

since

σ^{*} (x)

is not identifiable.) See Section 4 for details and the corresponding convergence proof.

We assume that

X_{t}

is sufficiently ergodic (to be concretely specified later in the paper) and that it has some well-behaved

π (d x)

as its unique invariant measure. As a general notation, if

h (x, θ)

is a generic

L^{1} (π)

function, then we define its average over

π (d x)

to be

\bar{h} (θ) = \int_{X} h (x, θ) π (d x) .

Let us set

g (x, θ) = \frac{1}{2} {∥ f (x, θ) - f^{*} (x) ∥}_{σ σ^{⊤}}^{2} = \frac{1}{2} ⟨ f (x, θ) - f^{*} (x), {(σ σ^{⊤})}^{- 1} (f (x, θ) - f^{*} (x)) ⟩ .

The gradient

\nabla_{θ} g (X_{t}, θ)

cannot be evaluated since

f^{*} (x)

is unknown. However,

d X_{t} = f^{*} (X_{t}) d t + σ d W_{t}

is a noisy estimate of

f^{*} (x) d t

, which leads to the algorithm 1.2. SGDCT follows a noisy descent direction along a continuous stream of data produced by

X_{t}

Heuristically, it is expected that

θ_{t}

will tend towards the minimum of the function

\bar{g} (θ) = \int_{X} g (x, θ) π (d x)

. The data

X_{t}

will be correlated over time, which complicates the mathematical analysis. This differs from the standard discrete-time version of stochastic gradient descent where the the data is usually considered to be i.i.d. at every step.

2. $1.1$ Literature Review

In this paper we show that if

α_{t}

is appropriately chosen then

\nabla \bar{g} (θ_{t}) \to 0

t \to \infty

with probability 1 (see Theorem 2.4). Results like this have been previously derived for stochastic gradient descent in discrete time; see [9] and [8]. 9] proves convergence in the absence of the

X

term. 8 , proves convergence of stochastic gradient descent in discrete time with the

X

process but requires stronger conditions than 9 .

Although stochastic gradient descent for discrete time has been extensively studied, stochastic gradient descent in continuous time has received relatively little attention. We refer readers to [8, 16] and [9] for a thorough review of the very large literature on stochastic gradient descent. There are also many algorithms which modify traditional stochastic gradient descent (stochastic gradient descent with momentum, Adagrad, RMSprop, etc.). For a review of these variants of stochastic gradient descent, see [15]. We mention below the prior work which is most relevant to our paper.

Our approach and assumptions required for convergence are most similar to [9], who prove convergence of discrete-time stochastic gradient descent in the absence of the

X

process. The presence of the

X

process is essential for considering a wide range of problems in continuous time, and showing convergence with its presence is considerably more difficult. The

X

term introduces correlation across times, and this correlation does not disappear as time tends to infinity. This makes it challenging to prove convergence in the continuoustime case. In order to prove convergence, we use an appropriate Poisson equation associated with

X

to describe the evolution of the parameters for large times.

21] proves, in a setting different than ours, convergence in

L^{2}

of projected stochastic gradient descent in discrete time for convex functions. In projected gradient descent, the parameters are projected back into an a priori chosen compact set. Therefore, the algorithm cannot hope to reach the minimum if the minimum is located outside of the chosen compact set. Of course, the compact set can be chosen to be very large for practical purposes. Our paper considers unconstrained stochastic gradient descent in continuous time and proves the almost sure convergence

\nabla \bar{g} (θ_{t}) \to 0

t \to \infty

taking into account the

X

component as well. We do not assume any stability conditions on

X

(except that it is ergodic with a unique invariant measure).

Another approach for proving convergence of discrete-time stochastic gradient descent is to show that the algorithm converges to the solution of an ODE which itself converges to a limiting point. This is the approach of [8]. See also [16. This method, sometimes called the "ODE method", requires the assumption that the iterates (i.e., the model parameters which are being learned) remain in a bounded set with probability one. It is unclear whether the ODE method of proof can be successfully used to show convergence for a continuoustime stochastic gradient descent scheme. In this paper we follow a potentially more straightforward method of proof by analyzing the speed of convergence to equilibrium with an appropriately chosen Poisson type of equation.

25] studies continuous-time stochastic mirror descent in a setting different than ours. In the framework of [25], the objective function is known. In this paper, we consider the statistical estimation of the unknown dynamics of a random process (i.e. the

X

process satisfying (1.1p).

Statisticians and financial engineers have actively studied parameter estimation of SDEs, although typically not with statistical learning or machine learning approaches. The likelihood function will usually be calculated from the entire observed path of

X

(i.e., batch optimization) and then maximized to find the maximum likelihood estimator (MLE). Unlike in this paper, the actual optimization procedure to maximize the likelihood function is often not analyzed.

Some relevant publications in the financial statistics literature include [1, 2], 7], and [14. 7] derives the likelihood function for continuously observed

X

. The MLE can be calculated via batch optimization. 1] and 2] consider the case where

X

is discretely observed and calculate MLEs via a batch optimization approach. [14] estimates parameters by a Bayesian approach. Readers are referred to [10, 17, 26] for thorough reviews of classical statistical inference methods for stochastic differential equations.

2.1. Applications of SGDCT

Continuous-time models are especially common in finance. Given a continuous stream of data, the stochastic gradient descent algorithm can be used to estimate unknown parameters or functions in SDE models for stocks, bonds, interest rates, and financial derivatives. Numerical analysis of SGDCT for two common financial models is included in Sections 5.1, 5.2, and 5.5. The first is the well-known Ornstein-Uhlenbeck (OU) process (for examples in finance, see [19], [20, 29], and [18]). The second is the multidimensional CIR process which is a common model for interest rates (for examples in finance, see [4, [22], [11, [5], and [12]).

Scientific and engineering models are also typically in continuous-time. There are often coefficients or functions in these models which are uncertain or unknown; stochastic gradient descent can be used to learn these model parameters from data. In Section 5, we study the numerical performance for two example applications: Burger's equation and the classic reinforcement learning problem of balancing a pole on a moving cart. Burger's equation is a widely used nonlinear partial differential equation which is important to fluid mechanics, acoustics, and aerodynamics.

A natural question is why use SGDCT versus a straightforward approach which (1) discretizes the continuous-time dynamics and then (2) applies traditional stochastic gradient descent. For some of the same reasons that scientific models have been largely developed in continuous time, it can be advantageous to develop continuous-time statistical learning for continuous-time models.

SGDCT allows for the application of numerical schemes of choice to the theoretically correct statistical learning equation for continuous-time models. This can lead to more accurate and more computationally efficient parameter updates. Numerical schemes are always applied to continuous-time dynamics and different numerical schemes may have different properties for different continuous-time models. A priori performing a discretization to the system dynamics and then applying a traditional discrete-time stochastic gradient descent scheme can result in a loss of accuracy. For example, there is no guarantee that (1) using a higherorder accurate scheme to discretize the system dynamics and then (2) applying traditional stochastic gradient descent will produce a statistical learning scheme which is higher-order accurate in time. Hence, it makes sense to first develop the continuous-time statistical learning equation, and then apply the higher-order accurate numerical scheme.

Besides model estimation, SGDCT can be used to solve continuous-time optimization problems, such as American options. We combine SGDCT with a deep neural network to solve American options in up to 100 dimensions (see Section 6). An alternative approach would be to discretize the dynamics and then use the Q-learning algorithm (traditional stochastic gradient descent applied to an approximation of the discrete HJB equation). However, Q-learning is biased while SGDCT is unbiased. Furthermore, in SDE models with Brownian motions, the Q-learning algorithm can blow up as the time step size

Δ

becomes small; see Section 6 for details.

The convergence issue with Q-learning highlights the importance of studying continuous-time algorithms for continuous-time models. It is of interest to show that (1) a discrete-time scheme converges to an appropriate continuous-time scheme as

Δ \to 0

and (2) the continuous-time scheme converges to the correct estimate as

t \to \infty

. These are important questions since any discrete scheme for a continuous-time model incurs some error proportional to

Δ

, and therefore

Δ

must be decreased to reduce error. It is also important to note that in some cases, such as Q-learning, computationally expensive terms in the discrete algorithm (such as expectations over high-dimensional spaces) may become much simpler expressions in the continuous-time scheme (differential operators).

2.2. Organization of Paper

The paper is organized into five main sections. Section 2 presents the assumption and the main theorem. In Section 3 we prove the main result of this paper for the convergence of continuous-time stochastic gradient descent. The extension of the stochastic gradient descent algorithm to the case of a variable diffusion coefficient function is described in Section 4 . Section 5 provides numerical analysis of SGDCT for model estimation in several applications. Section 6 discusses SGDCT for solving continuous-time optimization problems, particularly focusing on American options.

3. Assumptions and Main Result

Before presenting the main result of this paper, Theorem 2.4, let us elaborate on the standing assumptions. In regards to the learning rate

α_{t}

the standing assumption is

Condition 2.1. Assume that

\int_{0}^{\infty} α_{t} d t = \infty, \int_{0}^{\infty} α_{t}^{2} d t < \infty, \int_{0}^{\infty} | α_{s}^{'} | d s < \infty

and that there is a

p > 0

such that

lim_{t \to \infty} α_{t}^{2} t^{1 / 2 + 2 p} = 0

A standard choice for

α_{t}

that satisfies Condition 2.1 is

α_{t} = \frac{1}{C + t}

for some constant

0 < C < \infty

. Notice that the condition

\int_{0}^{\infty} | α_{s}^{'} | d s < \infty

follows immediately from the other two restrictions for the learning rate if it is chosen to be a monotonic function of

t

Let us next discuss the assumptions that we impose on

σ, f^{*} (x)

and

f (x, θ)

. Condition 2.2 guarantees uniqueness and existence of an invariant measure for the

X

process.

Condition 2.2. We assume that

σ σ^{⊤}

is non-degenerate bounded diffusion matrix and

lim_{| x | \to \infty} f^{*} (x) \cdot x = - \infty

In addition, with respect to

\nabla_{θ} f (x, θ)

we assume that

θ \in R^{n}

and we impose the following condition

Condition 2.3. 1. We assume that

\nabla_{θ} g (x, \cdot) \in C^{2} (R^{n})

for all

x \in X, \frac{\partial^{2} \nabla_{θ} g}{\partial x^{2}} \in C (X, R^{n}), \nabla_{θ} g (\cdot, θ) \in

C^{α} (X)

uniformly in

θ \in R^{n}

for some

α \in (0, 1)

and that there exist

K

and

q

such that

\sum_{i = 0}^{2} | \frac{\partial^{i} \nabla_{θ} g}{\partial θ^{i}} (x, θ) | \leq K (1 + | x |^{q}) .

For every $N > 0$ there exists a constant $C (N)$ such that for all $θ_{1}, θ_{2} \in R^{n}$ and $| x | \leq N$ , the diffusion coefficient $\nabla_{θ} f$ satisfies

| \nabla_{θ} f (x, θ_{1}) - \nabla_{θ} f (x, θ_{2}) | \leq C (N) | θ_{1} - θ_{2} | .

Moreover, there exists

K > 0

and

q > 0

such that

| \nabla_{θ} f (x, θ) | \leq K (1 + | x |^{q}) .

The function $f^{*} (x)$ is $C^{2 + α} (X)$ with $α \in (0, 1)$ . Namely, it has two derivatives in $x$ , with all partial derivatives being Hölder continuous, with exponent $α$ , with respect to $x$ .

Condition 2.3 allows one to control the ergodic behavior of the X process. As will be seen from the proof of the main convergence result Theorem 2.4. one needs to control terms of the form

\int_{0}^{t} α_{t} (\nabla \bar{g} (θ_{s}) - g (X_{s}, θ_{s})) d s

. Due to ergodicity of the

X

process one expects that such terms are small in magnitude and go to zero as

t \to \infty

. However, the speed at which they go to zero is what matters here. We treat such terms by rewriting them equivalently using appropriate Poisson type partial differential equations (PDE). Condition 2.3 guarantees that these Poisson equations have unique solutions that do not grow faster than polynomially in the

x

variable (see Theorem A.1 in Appendix A).

The main result of this paper is Theorem 2.4 .

Theorem 2.4. Assume that Conditions 2.1, 2.2 and 2.3 hold. Then we have that

lim_{t \to \infty} ∥ \nabla \bar{g} (θ_{t}) ∥ = 0, almost surely.

4. Proof of Theorem 2.4

We proceed in a spirit similar to that of [9]. However, apart from continuous versus discrete dynamics, one of the main challenges of the proof here is the presence of the ergodic

X

process. Let us consider an arbitrarily given

κ > 0

and

λ = λ (κ) > 0

to be chosen. Then set

σ_{0} = 0

and consider the cycles of random times

0 = σ_{0} \leq τ_{1} \leq σ_{1} \leq τ_{2} \leq σ_{2} \leq \dots

where for

k = 1, 2, \dots

\begin{aligned} τ_{k} & = inf {t > σ_{k - 1} : ∥ \nabla \bar{g} (θ_{t}) ∥ \geq κ}, \\ σ_{k} & = sup {t > τ_{k} : \frac{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}{2} \leq ∥ \nabla \bar{g} (θ_{s}) ∥ \leq 2 ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ for all s \in [τ_{k}, t] and \int_{τ_{k}}^{t} α_{s} d s \leq λ} . \end{aligned}

The purpose of these random times is to control the periods of time where

∥ \nabla \bar{g} (θ

∥

is close to zero and away from zero. Let us next define the random time intervals

J_{k} = [σ_{k - 1}, τ_{k})

and

I_{k} = [τ_{k}, σ_{k})

. Notice that for every

t \in J_{k}

we have

∥ \nabla \bar{g} (θ_{t}) ∥ < κ

Let us next consider some

η > 0

sufficiently small to be chosen later on and set

σ_{k, η} = σ_{k} + η

. Lemma 3.1 is crucial for the proof of Theorem 2.4.

Lemma 3.1. Assume that Conditions 2.1, 2.2 and 2.3 hold. Let us set

Γ_{k, η} = \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s .

Then, with probability one we have that

∥ Γ_{k, η} ∥ \to 0, as k \to \infty .

Proof. The idea is to use Theorem A.1 in order to get an equivalent expression for the term

Γ_{k, η}

that we seek to control.

Let us consider the function

G (x, θ) = \nabla_{θ} g (x, θ) - \nabla_{θ} \bar{g} (θ)

. Notice that by definition and due to Condition 2.3 the function

G (x, θ)

satisfies the centering condition A.1 of Theorem A.1 componentwise. So, the Poisson equation A.2 will have a unique smooth solution, denoted by

v (x, θ)

that grows at most polynomially in

x

. Let us apply Itô formula to the vector valued function

u (t, x, θ) = α_{t} v (x, θ)

. Doing so, we get for

i = 1, \dots, n

\begin{aligned} u_{i} (σ, X_{σ}, θ_{σ}) - u_{i} (τ, X_{τ}, θ_{τ}) & = \int_{τ}^{σ} \partial_{s} u_{i} (s, X_{s}, θ_{s}) d s + \int_{τ}^{σ} L_{x} u_{i} (s, X_{s}, θ_{s}) d s + \int_{τ}^{σ} L_{θ} u_{i} (s, X_{s}, θ_{s}) d s \\ + \int_{τ}^{σ} α_{s} tr [\nabla_{θ} f (X_{s}, θ_{s}) \nabla_{x} \nabla_{θ} u_{i} (s, X_{s}, θ_{s})] d s \\ + \int_{τ}^{σ} ⟨ \nabla_{x} u_{i} (s, X_{s}, θ_{s}), σ d W_{s} ⟩ + \int_{τ}^{σ} α_{s} ⟨ \nabla_{θ} u_{i} (s, X_{s}, θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩, \end{aligned}

where

L_{x}

and

L_{θ}

denote the infinitesimal generators for processes

X

and

θ

respectively.

Recall now that

v (x, θ)

is the solution to the given Poisson equation and that

u (s, x, θ) = α_{s} v (x, θ)

. Using these facts and rearranging the previous Itô formula, we get in vector notation

\begin{aligned} Γ_{k, η} = & \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s = \int_{τ_{k}}^{σ_{k, η}} L_{x} u (s, X_{s}, θ_{s}) d s \\ = & [α_{σ_{k, η}} v (X_{σ_{k, η}}, θ_{σ_{k, η}}) - α_{τ_{k}} v (X_{τ_{k}}, θ_{τ_{k}}) - \int_{τ_{k}}^{σ_{k, η}} \partial_{s} α_{s} v (X_{s}, θ_{s}) d s] \\ - \int_{τ_{k}}^{σ_{k, η}} α_{s} [L_{θ} v (X_{s}, θ_{s}) + α_{s} tr {[\nabla_{θ} f (X_{s}, θ_{s}) \nabla_{x_{i}} \nabla_{θ} v (X_{s}, θ_{s})]}_{i = 1}^{m}] d s \\ - \int_{τ_{k}}^{σ_{k, η}} α_{s} ⟨ \nabla_{x} v (X_{s}, θ_{s}), σ d W_{s} ⟩ - \int_{τ_{k}}^{σ_{k, η}} α_{s}^{2} ⟨ \nabla_{θ} v (X_{s}, θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ . \end{aligned}

The next step is to treat each term on the right hand side of (3.1) separately. For this purpose, let us first set

J_{t}^{(1)} = α_{t} sup_{s \in [0, t]} ∥ v (X_{s}, θ_{s}) ∥

By Theorem A.1 and Proposition 2 of [23] there is some

0 < K < \infty

(that may change from line to line below) and

0 < q < \infty

such that for

t

large enough

\begin{aligned} E {| J_{t}^{(1)} |}^{2} & \leq K α_{t}^{2} E [1 + sup_{s \in [0, t]} {∥ X_{s} ∥}^{q}] = K α_{t}^{2} [1 + \sqrt{t} \frac{E sup_{s \in [0, t]} {∥ X_{s} ∥}^{q}}{\sqrt{t}}] \\ \leq K α_{t}^{2} [1 + \sqrt{t}] \leq K α_{t}^{2} \sqrt{t} \end{aligned}

By Condition 2.1 let us consider

p > 0

such that

lim_{t \to \infty} α_{t}^{2} t^{1 / 2 + 2 p} = 0

and for any

δ \in (0, p)

define the event

A_{t, δ} = {J_{t}^{(1)} \geq t^{δ - p}}

. Then we have for

t

large enough such that

α_{t}^{2} t^{1 / 2 + 2 p} \leq 1

P (A_{t, δ}) \leq \frac{E {| J_{t}^{(1)} |}^{2}}{t^{2 (δ - p)}} \leq K \frac{α_{t}^{2} t^{1 / 2 + 2 p}}{t^{2 δ}} \leq K \frac{1}{t^{2 δ}}

The latter implies that

\sum_{n \in N} P (A_{2^{n}, δ}) < \infty

Therefore, by Borel-Cantelli lemma we have that for every

δ \in (0, p)

there is a finite positive random variable

d (ω)

and some

n_{0} < \infty

such that for every

n \geq n_{0}

one has

J_{2^{n}}^{(1)} \leq \frac{d (ω)}{2^{n (p - δ)}}

Thus for

t \in [2^{n}, 2^{n + 1})

and

n \geq n_{0}

one has for some finite constant

K < \infty

J_{t}^{(1)} \leq K α_{2^{n + 1}} sup_{s \in (0, 2^{n + 1}]} ∥ v (X_{s}, θ_{s}) ∥ \leq K \frac{d (ω)}{2^{(n + 1) (p - δ)}} \leq K \frac{d (ω)}{t^{p - δ}} .

The latter display then guarantees that for

t \geq 2^{n_{0}}

we have with probability one

J_{t}^{(1)} \leq K \frac{d (ω)}{t^{p - δ}} \to 0, as t \to \infty

Next we consider the term

J_{t, 0}^{(2)} = \int_{0}^{t} ∥ α_{s}^{'} v (X_{s}, θ_{s}) + α_{s} (L_{θ} v (X_{s}, θ_{s}) + α_{s} tr {[\nabla_{θ} f (X_{s}, θ_{s}) \nabla_{x_{i}} \nabla_{θ} v (X_{s}, θ_{s})]}_{i = 1}^{m}) ∥ d s .

By the bounds of Theorem A.1 we see that there are constants

0 < K < \infty

(that may change from line to line) and

0 < q < \infty

such that

\begin{aligned} sup_{t > 0} E | J_{t, 0}^{(2)} | & \leq K \int_{0}^{\infty} (| α_{s}^{'} | + α_{s}^{2}) (1 + E {∥ X_{s} ∥}^{q}) d s \\ \leq K \int_{0}^{\infty} (| α_{s}^{'} | + α_{s}^{2}) d s \\ \leq K \end{aligned}

The first inequality follows by Theorem A.1. the second inequality follows by Proposition 1 in [23. and the third inequality follows by Condition 2.1.

The latter display implies that there is a finite random variable

{\bar{J}}_{\infty, 0}^{(2)}

such that

J_{t, 0}^{(2)} \to {\bar{J}}_{\infty, 0}^{(2)}, as t \to \infty with probability one.

The last term that we need to consider is the martingale term

J_{t, 0}^{(3)} = \int_{0}^{t} α_{s} ⟨ \nabla_{x} v (X_{s}, θ_{s}), σ d W_{s} ⟩ + \int_{0}^{t} α_{s}^{2} ⟨ \nabla_{θ} v (X_{s}, θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ .

Notice that the Burkholder-Davis-Gundy inequality and the bounds of Theorem A.1 (doing calculations similar to the ones for the term

J_{t, 0}^{(2)})

give us that for some finite constant

K < \infty

, we have

sup_{t > 0} E {| J_{t, 0}^{(3)} |}^{2} \leq K \int_{0}^{\infty} α_{s}^{2} d s < \infty

that

Thus, by Doob's martingale convergence theorem there is a square integrable random variable

{\bar{J}}_{\infty, 0}^{(3)}

such

J_{t, 0}^{(3)} \to {\bar{J}}_{\infty, 0}^{(3)}, as t \to \infty both almost surely and in L^{2} .

Let us now go back to 3.1 . Using the terms

J_{t}^{(1)}, J_{t, 0}^{(2)}

and

J_{t, 0}^{(3)}

we can write

∥ Γ_{k, η} ∥ \leq J_{σ_{k, η}}^{(1)} + J_{τ_{k}}^{(1)} + J_{σ_{k, η}, τ_{k}}^{(2)} + ∥ J_{σ_{k, η}, τ_{k}}^{(3)} ∥

The last display together with 3.2 , (3.3) and 3.4 imply the statement of the lemma.

Lemma 3.2. Assume that Conditions 2.1, 2.2 and 2.3 hold. Choose

λ > 0

such that for a given

κ > 0

, one has

3 λ + \frac{λ}{4 κ} = \frac{1}{2 L_{\nabla \bar{g}}}

, where

L_{\nabla \bar{g}}

is the Lipschitz constant of

\nabla \bar{g}

. For

k

large enough and for

η > 0

small enough (potentially random depending on

k

), one has

\int_{τ_{k}}^{σ_{k, η}} α_{s} d s > λ

. In addition we also have

\frac{λ}{2} \leq \int_{τ_{k}}^{σ_{k}} α_{s} d s \leq λ

with probability one.

Proof. Let us define the random variable

R_{s} = \sum_{k \geq 1} ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ 1_{s \in I_{k}} + κ 1_{s \in [0, \infty) ∖ ⋃_{k \geq 1} I_{k}} .

Then, for any

s \in R

we have

∥ \nabla \bar{g} (θ_{s}) ∥ / R_{s} \leq 2

We proceed with an argument via contradiction. In particular let us assume that

\int_{τ_{k}}^{σ_{k, η}} α_{s} d s \leq λ

and let us choose arbitrarily some

ϵ > 0

such that

ϵ \leq λ / 8

Let us now make some remarks that are independent of the sign of

\int_{τ_{k}}^{σ_{k, η}} α_{s} d s - λ

. Due to the summability condition

\int_{0}^{\infty} α_{t}^{2} d t < \infty, \frac{κ}{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥} \leq 1

and Conditions 2.1 and 2.3 , we have that

sup_{t > 0} E {| \int_{0}^{t} α_{s} \frac{κ}{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥} \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} |}^{2} < \infty

Hence, the martingale convergence theorem applies to the martingale

\int_{0}^{t} α_{s} \frac{κ}{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥} \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s}

. This means that there exists a square integrable random variable

M

such that

\int_{0}^{t} α_{s} \frac{κ}{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥} \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} \to

M

both almost surely and in

L^{2}

. This means that for the given

ϵ > 0

there is

k

large enough such that

∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} \frac{κ}{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥} \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ∥ < ϵ

almost surely.

Let us also assume that for the given

k, η

is so small such that for any

s \in [τ_{k}, σ_{k, η}]

one has

∥ \nabla \bar{g} (θ_{s}) ∥ \leq

3 ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥

Then, we obtain the following

\begin{aligned} ∥ θ_{σ_{k, η}} - θ_{τ_{k}} ∥ = & ∥ - \int_{τ_{k}}^{σ_{k, η}} α_{s} \nabla_{θ} g (X_{s}, θ_{s}) d s + \int_{τ_{k}}^{σ_{k, η}} α_{s} \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ∥ \\ = & ∥ - \int_{τ_{k}}^{σ_{k, η}} α_{s} \nabla_{θ} \bar{g} (θ_{s}) d s - \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s + \int_{τ_{k}}^{σ_{k, η}} α_{s} \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ∥ \\ \leq & \int_{τ_{k}}^{σ_{k, η}} α_{s} ∥ \nabla \bar{g} (θ_{s}) ∥ d s + ∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s ∥ + ∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ∥ \\ \leq & 3 ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} d s + ∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s ∥ \\ + \frac{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}{κ} ∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} \frac{κ}{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥} \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ∥ \\ \leq & 3 ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ λ + ∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s ∥ + \frac{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}{κ} ϵ \\ \leq & 3 ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ λ + ∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s ∥ + \frac{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}{κ} λ / 8 \\ = & ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ [3 λ + \frac{λ}{8 κ}] + ∥ \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s ∥ . \end{aligned}

Let us next bound appropriately the Euclidean norm of the vector-valued random variable

Γ_{k, η} = \int_{τ_{k}}^{σ_{k, η}} α_{s} (\nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s})) d s .

By Lemma 3.1 we have that for the same

0 < ϵ < λ / 8

that was chosen before there is

k

large enough such that almost surely

∥ Γ_{k, η} ∥ \leq ϵ \leq λ / 8 .

Hence, using also the fact that

\frac{κ}{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥} \leq 1

we obtain

∥ θ_{σ_{k, η}} - θ_{τ_{k}} ∥ \leq ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ [3 λ + \frac{λ}{4 κ}] = ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ \frac{1}{2 L_{\nabla \bar{g}}} .

The latter then implies that we should have

∥ \nabla \bar{g} (θ_{σ_{k, η}}) - \nabla \bar{g} (θ_{τ_{k}}) ∥ \leq L_{\nabla \bar{g}} ∥ θ_{σ_{k, η}} - θ_{τ_{k}} ∥ \leq \frac{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}{2} .

The latter statement will then imply that

\frac{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}{2} \leq ∥ \nabla \bar{g} (θ_{σ_{k, η}}) ∥ \leq 2 ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ .

But then we would necessarily have that

\int_{τ_{k}}^{σ_{k, η}} α_{s} d s > λ

, since otherwise

σ_{k, η} \in [τ_{k}, σ_{k}]

which is impossible.

Next we move on to prove the second statement of the lemma. By definition we have

\int_{τ_{k}}^{σ_{k}} α_{s} d s \leq λ

. So it remains to show that

\frac{λ}{2} \leq \int_{τ_{k}}^{σ_{k}} α_{s} d s

. Since we know that

\int_{τ_{k}}^{σ_{k, η}} α_{s} d s > λ

and because for

k

large enough and

η

small enough one should have

\int_{σ_{k}}^{σ_{k, η}} α_{s} d s \leq λ / 2

, we obtain that

\int_{τ_{k}}^{σ_{k}} α_{s} d s \geq λ - \int_{σ_{k}}^{σ_{k, η}} α_{s} d s \geq λ - λ / 2 = λ / 2

concluding the proof of the lemma.

Lemma 3.3 shows that the function

\bar{g}

and its first two derivatives are uniformly bounded in

θ

Lemma 3.3. Assume Conditions 2.1, 2.2 and 2.3. For any

q > 0

, there is a constant

K

such that

\int_{X} (1 + | x |^{q}) π (d x) \leq C

In addition we also have that there is a constant

C < \infty

such that

\sum_{i = 0}^{2} ∥ \nabla_{θ}^{i} \bar{g} (θ) ∥ \leq C

Proof. By Theorem 1 in [24, the density

μ

of the measure

π

admits, for any

p

, a constant

C_{p}

such that

μ (x) \leq \frac{C_{p}}{1 + | x |^{p}}

. Choosing

p

large enough that

\int_{X} \frac{1 + | x |^{q}}{1 + | x |^{p}} d y < \infty

, we then obtain

\int_{X} (1 + | x |^{q}) π (d x) \leq \int_{X} C_{p} \frac{1 + | x |^{q}}{1 + | x |^{p}} d x \leq C .

concluding the proof of the first statement of the lemma. Let us now focus on the second part of the lemma. We only prove the claim for

i = 0

, since due to the bounds in Condition 2.3 the proof for

i = 1, 2

is the same. By Condition 2.3 and by the first part of the lemma, we have that there exist constants

0 < q, K, C < \infty

such that

\bar{g} (θ) = \int_{X} \frac{1}{2} {∥ f (x, θ) - f^{*} (x) ∥}^{2} π (d x) \leq K \int_{X} (1 + | x |^{q}) π (d x) \leq C,

concluding the proof of the lemma.

Our next goal is to show that if the index

k

is large enough, then

\bar{g}

decreases, in the sense of Lemma 3.4

Lemma 3.4. Assume Conditions 2.1. 2.2 and 2.3. Suppose that there are an infinite number of intervals

I_{k} = [τ_{k}, σ_{k})

. There is a fixed constant

γ = γ (κ) > 0

such that for

k

large enough, one has

\bar{g} (θ_{σ_{k}}) - \bar{g} (θ_{τ_{k}}) \leq - γ .

Proof. By Itô's formula we have that

\begin{aligned} \bar{g} (θ_{σ_{k}}) - \bar{g} (θ_{τ_{k}}) & = - \int_{τ_{k}}^{σ_{k}} α_{s} {∥ \nabla \bar{g} (θ_{s}) ∥}^{2} d s + \int_{τ_{k}}^{σ_{k}} α_{s} ⟨ \nabla \bar{g} (θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ \\ + \int_{τ_{k}}^{σ_{k}} \frac{α_{s}^{2}}{2} tr [(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1}) {(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1})}^{⊤} \nabla_{θ} \nabla_{θ} \bar{g} (θ_{s})] d s \\ + \int_{τ_{k}}^{σ_{k}} α_{s} ⟨ \nabla_{θ} \bar{g} (θ_{s}), \nabla_{θ} \bar{g} (θ_{s}) - \nabla_{θ} g (X_{s}, θ_{s}) ⟩ d s \\ = Θ_{1, k} + Θ_{2, k} + Θ_{3, k} + Θ_{4, k} . \end{aligned}

Let's first consider

Θ_{1, k}

. Notice that for all

s \in [τ_{k}, σ_{k}]

one has

\frac{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}{2} \leq ∥ \nabla \bar{g} (θ_{s}) ∥ \leq 2 ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥

. Hence, for sufficiently large

k

, we have the upper bound:

Θ_{1, k} = - \int_{τ_{k}}^{σ_{k}} α_{s} {∥ \nabla \bar{g} (θ_{s}) ∥}^{2} d s \leq - \frac{{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}^{2}}{4} \int_{τ_{k}}^{σ_{k}} α_{s} d s \leq - \frac{{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}^{2}}{8} λ,

since Lemma 3.1 proved that

\int_{τ_{k}}^{σ_{k}} α_{s} d s \geq \frac{λ}{2}

for sufficiently large

k

We next address

Θ_{2, k}

and show that it becomes small as

k \to \infty

. First notice that we can trivially write

\begin{aligned} Θ_{2, k} & = \int_{τ_{k}}^{σ_{k}} α_{s} ⟨ \nabla \bar{g} (θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ = ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ \int_{τ_{k}}^{σ_{k}} α_{s} ⟨ \frac{\nabla \bar{g} (θ_{s})}{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}, \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ \\ = ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ \int_{τ_{k}}^{σ_{k}} α_{s} ⟨ \frac{\nabla \bar{g} (θ_{s})}{R_{s}}, \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ . \end{aligned}

By Condition 2.3 and Itô isometry we have

\begin{aligned} sup_{t > 0} E {| \int_{0}^{t} α_{s} ⟨ \frac{\nabla \bar{g} (θ_{s})}{R_{s}}, \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ |}^{2} & \leq 4 E \int_{0}^{\infty} α_{s}^{2} {∥ \nabla_{θ} f (X_{s}, θ_{s}) ∥}^{2} d s \\ \leq K \int_{0}^{\infty} α_{s}^{2} (1 + E {∥ X_{s} ∥}^{q}) d s < \infty \end{aligned}

where

R_{s}

is defined via (3.5). Hence, by Doob's martingale convergence theorem there is a square integrable random variable

M

such that

\int_{0}^{t} α_{s} ⟨ \frac{\nabla \bar{g} (θ_{s})}{R_{s}}, \nabla_{θ} f (X_{s}, θ_{s}) d W_{s} ⟩ \to M

both almost surely and in

L^{2}

. The latter statement implies that for a given

ϵ > 0

there is

k

large enough such that almost surely

Θ_{2, k} \leq ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ ϵ .

We now consider

Θ_{3, k}

\begin{aligned} sup_{t > 0} E ∥ \int_{0}^{t} \frac{α_{s}^{2}}{2} tr [(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1}) {(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1})}^{⊤} \nabla_{θ} \nabla_{θ} \bar{g} (θ_{s})] d s ∥ \\ \leq & C \int_{0}^{\infty} \frac{α_{s}^{2}}{2} E (1 + {∥ X_{s} ∥}^{q}) d s < \infty, \end{aligned}

where we have used Condition 2.3 and Lemma 3.3 . Bound 3.7 implies that

\int_{0}^{\infty} \frac{α_{s}^{2}}{2} tr [(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1}) {(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1})}^{⊤} \nabla_{θ} \nabla_{θ} \bar{g} (θ_{s})] d s

is finite almost surely, which in turn implies that there is a finite random variable

Θ_{3}^{\infty}

such that

\int_{0}^{t} \frac{α_{s}^{2}}{2} tr [(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1}) {(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1})}^{⊤} \nabla_{θ} \nabla_{θ} \bar{g} (θ_{s})] d s \to Θ_{3}^{\infty} as t \to \infty

with probability one. Since

Θ_{3}^{\infty}

is finite,

\int_{τ_{k}}^{σ_{k}} \frac{α_{s}^{2}}{2} tr [(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1}) {(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1})}^{⊤} \nabla_{θ} \nabla_{θ} \bar{g} (θ_{s})] d s \to 0

k \to \infty

with probability one.

Finally, we address

Θ_{4, k}

. Let us consider the function

G (x, θ) = ⟨ \nabla_{θ} \bar{g} (θ), \nabla_{θ} g (x, θ) - \nabla_{θ} \bar{g} (θ) ⟩

. The function

G (x, θ)

satisfies the centering condition A.1 of Theorem A.1. Therefore, the Poisson equation A.2 with right hand side

G (x, θ)

will have a unique smooth solution, say

v (x, θ)

, that grows at most polynomially in

x

. Let us apply Itô formula to the function

u (t, x, θ) = α_{t} v (x, θ)

that is solution to this Poisson equation.

\begin{aligned} u (σ, X_{σ}, θ_{σ}) - u (τ, X_{τ}, θ_{τ}) & = \int_{τ}^{σ} \partial_{s} u (s, X_{s}, θ_{s}) d s + \int_{τ}^{σ} L_{x} u (s, X_{s}, θ_{s}) d s + \int_{τ}^{σ} L_{θ} u (s, X_{s}, θ_{s}) d s \\ + \int_{τ}^{σ} α_{s} tr [\nabla_{θ} f (X_{s}, θ_{s}) \nabla_{x} \nabla_{θ} u (s, X_{s}, θ_{s})] d s \\ + \int_{τ}^{σ} ⟨ \nabla_{x} u (s, X_{s}, θ_{s}), σ d W_{s} ⟩ + \int_{τ}^{σ} α_{s} ⟨ \nabla_{θ} u (s, X_{s}, θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ . \end{aligned}

Rearranging the previous Itô formula yields

\begin{aligned} Θ_{4, k} = & \int_{τ_{k}}^{σ_{k}} α_{s} ⟨ \nabla_{θ} \bar{g} (θ_{t}), \nabla_{θ} g (X_{s}, θ_{s}) - \nabla_{θ} \bar{g} (θ_{s}) ⟩ d s = \int_{τ_{k}}^{σ_{k}} L_{x} u (s, X_{s}, θ_{s}) d s \\ = & [α_{σ_{k}} v (X_{σ_{k}}, θ_{σ_{k}}) - α_{τ_{k}} v (X_{τ_{k}}, θ_{τ_{k}}) - \int_{τ_{k}}^{σ_{k}} \partial_{s} α_{s} v (X_{s}, θ_{s}) d s] \\ - \int_{τ_{k}}^{σ_{k}} α_{s} [L_{θ} v (X_{s}, θ_{s}) + α_{s} tr [\nabla_{θ} f (X_{s}, θ_{s}) \nabla_{x} \nabla_{θ} v (X_{s}, θ_{s})]] d s \\ - \int_{τ_{k}}^{σ_{k}} α_{s} ⟨ \nabla_{x} v (X_{s}, θ_{s}), σ d W_{s} ⟩ - \int_{τ_{k}}^{σ_{k}} α_{s} ⟨ \nabla_{θ} v (X_{s}, θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ . \end{aligned}

Following the exact same steps as in the proof of Lemma 3.1 gives us that

lim_{k \to \infty} ∥ Θ_{4, k} ∥ \to 0

almost surely.

We now return to

\bar{g} (θ_{σ_{k}}) - \bar{g} (θ_{τ_{k}})

and provide an upper bound which is negative. For sufficiently large

k

, we have that:

\begin{aligned} \bar{g} (θ_{σ_{k}}) - \bar{g} (θ_{τ_{k}}) & \leq - \frac{{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}^{2}}{8} λ + ∥ Θ_{2, k} ∥ + ∥ Θ_{3, k} ∥ + ∥ Θ_{4, k} ∥ \\ \leq - \frac{{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}^{2}}{8} λ + ∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ ϵ + ϵ + ϵ . \end{aligned}

Choose

ϵ = min {\frac{λ κ^{2}}{32}, \frac{λ}{32}}

. On the one hand, if

∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ \geq 1

\begin{aligned} \bar{g} (θ_{σ_{k}}) - \bar{g} (θ_{τ_{k}}) & \leq - \frac{{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}^{2}}{8} λ + {∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}^{2} ϵ + ϵ + ϵ \\ \leq - 3 \frac{{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}^{2}}{32} λ + 2 ϵ \leq - 3 \frac{κ^{2}}{32} λ + 2 \frac{κ^{2}}{32} λ \leq - \frac{κ^{2}}{32} λ . \end{aligned}

On the other hand, if

∥ \nabla \bar{g} (θ_{τ_{k}}) ∥ \leq 1

, then

\begin{aligned} \bar{g} (θ_{σ_{k}}) - \bar{g} (θ_{τ_{k}}) & \leq - \frac{{∥ \nabla \bar{g} (θ_{τ_{k}}) ∥}^{2}}{8} λ + ϵ + ϵ + ϵ \\ \leq - \frac{4 κ^{2}}{32} λ + 3 ϵ \leq - 4 \frac{κ^{2}}{32} λ + 3 \frac{κ^{2}}{32} λ \leq - \frac{κ^{2}}{32} λ . \end{aligned}

Finally, let

γ = \frac{κ^{2}}{32} λ

and the proof of the lemma is complete.

Lemma 3.5. Assume Conditions 2.1, 2.2 and 2.3. Suppose that there are an infinite number of intervals

I_{k} = [τ_{k}, σ_{k})

. There is a fixed constant

γ_{1} < γ

such that for

k

large enough,

\bar{g} (θ_{τ_{k}}) - \bar{g} (θ_{σ_{k - 1}}) \leq γ_{1} .

Proof. First, recall that

∥ \nabla \bar{g} (θ_{t}) ∥ \leq κ

for

t \in J_{k} = [σ_{k - 1}, τ_{k}]

. Similar to before, we have that:

\begin{aligned} \bar{g} (θ_{τ_{k}}) - \bar{g} (θ_{σ_{k - 1}}) & = - \int_{σ_{k - 1}}^{τ_{k}} α_{s} {∥ \nabla \bar{g} (θ_{s}) ∥}^{2} d s + \int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \nabla \bar{g} (θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ \\ + \int_{σ_{k - 1}}^{τ_{k}} \frac{α_{s}^{2}}{2} tr [(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1}) {(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1})}^{⊤} \nabla_{θ}^{2} \bar{g} (θ_{s})] d s \\ + \int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \nabla_{θ} \bar{g} (θ_{s}), \nabla_{θ} \bar{g} (θ_{s}) - \nabla_{θ} g (X_{s}, θ_{s}) ⟩ d s \\ \leq \int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \nabla \bar{g} (θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ \\ + \int_{σ_{k - 1}}^{τ_{k}} \frac{α_{s}^{2}}{2} tr [(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1}) {(\nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1})}^{⊤} \nabla_{θ}^{2} \bar{g} (θ_{s})] d s \\ + \int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \nabla_{θ} \bar{g} (θ_{s}), \nabla_{θ} \bar{g} (θ_{s}) - \nabla_{θ} g (X_{s}, θ_{s}) ⟩ d s . \end{aligned}

The right hand side (RHS) of equation (3.8) converges almost surely to 0 as

k \to \infty

as a consequence of similar arguments as given in Lemma 3.4 Indeed, the treatment of the second and third terms on the RHS of (3.8) are exactly the same as in Lemma 3.4. It remains to show that the first term on the RHS of (3.8) converges almost surely to 0 as

k \to \infty

\begin{aligned} \int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \nabla \bar{g} (θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ & = ∥ \nabla \bar{g} (θ_{σ_{k - 1}}) ∥ \int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \frac{\nabla \bar{g} (θ_{s})}{∥ \nabla \bar{g} (θ_{σ_{k - 1}}) ∥}, \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ \\ = ∥ \nabla \bar{g} (θ_{σ_{k - 1}}) ∥ \int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \frac{\nabla \bar{g} (θ_{s})}{R_{s}}, \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ . \end{aligned}

As shown in Lemma 3.4.

\int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \frac{\nabla \bar{g} (θ_{s})}{R_{s}}, \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ \to 0

k \to \infty

almost surely. Finally, note that

∥ \nabla \bar{g} (θ_{σ_{k - 1}}) ∥ \leq κ

(except when

σ_{k - 1} = τ_{k}

, in which case the interval

J_{k}

is length 0 and hence the integral 3.9 over

J_{k}

is 0

)

. Then,

\int_{σ_{k - 1}}^{τ_{k}} α_{s} ⟨ \nabla \bar{g} (θ_{s}), \nabla_{θ} f (X_{s}, θ_{s}) σ^{- 1} d W_{s} ⟩ \to 0

k \to \infty

almost surely.

Therefore, with probability one,

\bar{g} (θ_{τ_{k}}) - \bar{g} (θ_{σ_{k - 1}}) \leq γ_{1} < γ

for sufficiently large

k

Proof of Theorem 2.4. Choose a

κ > 0

. First, consider the case where there are a finite number of times

τ_{k}

. Then, there is a finite

T

such that

∥ \nabla \bar{g} (θ_{t}) ∥ < κ

for

t \geq T

. Now, consider the other case where there are an infinite number of times

τ_{k}

and use Lemmas 3.4 and 3.5 . With probability one,

\begin{aligned} \bar{g} (θ_{σ_{k}}) - \bar{g} (θ_{τ_{k}}) & \leq - γ = - \frac{κ^{2}}{32} λ, \\ \bar{g} (θ_{τ_{k}}) - \bar{g} (θ_{σ_{k - 1}}) & \leq γ_{1} < γ, \end{aligned}

for sufficiently large

k

. Choose a

K

such that 3.10 holds for

k \geq K

. This leads to:

\bar{g} (θ_{τ_{n + 1}}) - \bar{g} (θ_{τ_{K}}) = \sum_{k = K}^{n} [\bar{g} (θ_{σ_{k}}) - \bar{g} (θ_{τ_{k}}) + \bar{g} (θ_{τ_{k + 1}}) - \bar{g} (θ_{σ_{k}})] \leq \sum_{k = K}^{n} (- γ + γ_{1}) < 0 .

Let

n \to \infty

and then

\bar{g} (θ_{τ_{n + 1}}) \to - \infty

. However, we also have that by definition

\bar{g} (θ) \geq 0

. This is a contradiction, and therefore almost surely there are a finite number of times

τ_{k}

Consequently, there exists a finite time

T

(possibly random) such that almost surely

∥ \nabla \bar{g} (θ_{t}) ∥ < κ

for

t \geq T

. Since the original

κ > 0

was arbitrarily chosen, this shows that

∥ \nabla \bar{g} (θ_{t}) ∥ \to 0

t \to \infty

almost surely.

5. Estimating the Coefficient Function of the Diffusion Term and Generalizations

We consider a diffusion

X_{t} \in X = R^{m}

d X_{t} = f^{*} (X_{t}) d t + σ^{*} (X_{t}) d W_{t} .

The goal is to statistically estimate a model

f (x, θ)

for

f^{*} (x)

as well as a model

σ (x, ν) σ^{⊤} (x, ν)

for the diffusion coefficient

σ^{*} (x) σ^{*, ⊤} (x)

where

θ \in R^{n}

and

ν \in R^{k}

W_{t} \in R^{m}

is a standard Brownian motion and

σ^{*} (\cdot) \in R^{m \times m}

. The functions

f (x, θ), σ (x, ν), f^{*} (x)

, and

σ^{*} (x)

may be non-convex.

The stochastic gradient descent update in continuous time follows the stochastic differential equations:

\begin{aligned} d θ_{t} & = α_{t} \nabla_{θ} f (X_{t}, θ_{t}) [d X_{t} - f (X_{t}, θ_{t}) d t], \\ d ν_{t} & = α_{t} \sum_{i, j}^{m} \nabla_{ν} ({(σ (X_{t}, ν_{t}) σ^{⊤} (X_{t}, ν_{t}))}_{i, j}) [d {⟨ X_{t}, X_{t} ⟩}_{i, j} - {(σ (X_{t}, ν_{t}) σ^{⊤} (X_{t}, ν_{t}))}_{i, j} d t], \end{aligned}

where

⟨ X_{t}, X_{t} ⟩ \in R^{m \times m}

is the quadratic variation matrix of

X

. Since we observe the path of

X_{t}

, we also observe the path of the quadratic variation

⟨ X_{t}, X_{t} ⟩

Let us set:

\begin{aligned} g (x, θ) & = \frac{1}{2} {∥ f (x, θ) - f^{*} (x) ∥}^{2} \\ w (x, ν) & = \frac{1}{2} {∥ σ (x, ν) σ^{⊤} (x, ν) - σ^{*} (x) σ^{*, ⊤} (x) ∥}^{2} . \end{aligned}

We assume that

σ^{*} (x)

is such that the process

X_{t}

is ergodic with a unique invariant measure (for example one may assume that it is non-degenerate, i.e., bounded away from zero and bounded by above). In addition, we assume that

w (x, ν)

satisfies the same assumptions as

g (x, θ)

does in Condition 2.3 .

From the previous results in Section

3, lim_{t \to \infty} ∥ \nabla \bar{g} (θ_{t}) ∥ = 0

t \to \infty

with probability 1 . Let's study the convergence of the stochastic gradient descent algorithm 4.2 for

ν_{t}

. By Itô's formula,

\bar{w} (ν_{σ}) - \bar{w} (ν_{τ}) = - \int_{τ}^{σ} α_{s} {∥ \nabla \bar{w} (ν_{s}) ∥}^{2} d s + \int_{τ}^{σ} α_{s} ⟨ \nabla_{ν} \bar{w} (ν_{s}), \nabla_{ν} \bar{w} (ν_{s}) - \nabla_{ν} w (X_{s}, ν_{s}) ⟩ d s

Applying exactly the same procedure as in Section

3, lim_{t \to \infty} ∥ \nabla \bar{w} (ν_{t}) ∥ = 0

t \to \infty

with probability 1 . We omit the details as the proof is exactly the same as in Section 3.

Notice also that

σ^{*} (x)

is not identifiable; for example,

X_{t}

has the same distribution under the diffusion coefficient

- σ^{*} (x)

. Only

σ^{*} (x) σ^{*, ⊤} (x)

is identifiable. We are therefore essentially estimating a model

σ (x, ν) σ^{⊤} (x, ν)

for

σ^{*} (x) σ^{*, ⊤} (x)

We close this section with the following remark.

Remark 4.1. The proof of Theorem 2.4 makes it clear that if appropriate assumptions on

\nabla_{θ} f

and

\nabla_{g} θ

are made such that

sup_{t > 0} E | θ_{t}^{q} | < C

for appropriate

0 < q, C < \infty

, then one can relax Condition 2.3 on

\nabla_{θ} g

to allow at least linear growth with respect to

θ

6. Model Estimation: Numerical Analysis

We implement SGDCT for several applications and numerically analyze the convergence. Section 5.1 studies continuous-time stochastic gradient descent for the Ornstein-Uhlenbeck process, which is widely used in finance, physics, and biology. Section 5.2 studies the multidimensional Ornstein-Uhlenbeck process. Section 5.3 estimates the diffusion coefficient in Burger's equation with continuous-time stochastic gradient descent. Burger's equation is a widely-used nonlinear partial differential equation which is important to fluid mechanics, acoustics, and aerodynamics. Burger's equation is extensively used in engineering. In Section 5.4. we show how SGDCT can be used for reinforcement learning. In the final example, the drift and volatility functions for the multidimensional CIR process are estimated. The CIR process is widely used in financial modeling.

6.1. Ornstein-Uhlenbeck process

The Ornstein-Uhlenbeck (OU) process

X_{t} \in R

satisfies the stochastic differential equation:

d X_{t} = c (m - X_{t}) d t + d W_{t} .

We use continuous-time stochastic gradient descent to learn the parameters

θ = (c, m) \in R^{2}

For the numerical experiments, we use an Euler scheme with a time step of

10^{- 2}

. The learning rate is

α_{t} = min (α, α / t)

with

α = 10^{- 2}

. We simulate data from (5.1) for a particular

θ^{*}

and the stochastic gradient descent attempts to learn a parameter

θ_{t}

which fits the data well.

θ_{t}

is the statistical estimate for

θ^{*}

at time

t

. If the estimation is accurate,

θ_{t}

should of course be close to

θ^{*}

. This example can be placed in the form of the original class of equations (1.1) by setting

f (x, θ) = c (m - x)

and

f^{*} (x) = f (x, θ^{*})

We study 10,500 cases. For each case, a different

θ^{*}

is generated uniformly at random in the range

[1, 2] \times [1, 2]

. For each case, we solve for the parameter

θ_{t}

over the time period

[0, T]

for

T = 10^{6}

. To summarize: - For cases

n = 1

to 10,500

Generate a random $θ^{*}$ in $[1, 2] \times [1, 2]$
Simulate a single path of $X_{t}$ given $θ^{*}$ and simultaneously solve for the path of $θ_{t}$ on $[0, T]$

The accuracy of

θ_{t}

at times

t = 10^{2}, 10^{3}, 10^{4}, 10^{5}

, and

10^{6}

is reported in Table 1 . Figures 1 and 2 plot the mean error in percent and mean squared error (MSE) against time. In the table and figures, the "error" is

| θ_{t}^{n} - θ^{*, n} |

where

n

represents the

n

-th case. The "error in percent" is

100 \times \frac{| θ_{t}^{n} - θ^{*, n} |}{| θ^{*, n} |}

. The "mean error in percent" is the average of these errors, i.e.

\frac{100}{N} \sum_{n = 1}^{N} \frac{| θ_{t}^{n} - θ^{*, n} |}{| θ^{*, n} |}

Figure 1: Mean error in percent plotted against time. Time is in log scale.

Figure 2: Mean squared error plotted against time. Time is in log scale.

Error/Time	$10^{2}$	$10^{3}$	$10^{4}$	$10^{5}$	$10^{6}$
Maximum Error	.604	.2615	.0936	.0349	.0105
$99 %$ quantile of error	.368	.140	.0480	.0163	.00542
$99.9 %$ quantile of error	.470	.1874	.0670	.0225	.00772
Mean squared error	$1.92 \times 10^{- 2}$	$2.28 \times 10^{- 3}$	$2.52 \times 10^{- 4}$	$2.76 \times 10^{- 5}$	$2.90 \times 10^{- 6}$
Mean Error in percent	7.37	2.497	0.811	0.264	0.085
Maximum error in percent	59.92	20.37	5.367	1.79	0.567
$99 %$ quantile of error in percent	25.14	9.07	3.05	1.00	0.323
$99.9 %$ quantile of error in percent	34.86	12.38	4.12	1.30	0.432

Table 1: Error at different times for the estimate

θ_{t}

θ^{*}

across 10,500 cases. The "error" is

| θ_{t}^{n} - θ^{*, n} |

where

n

represents the

n

-th case. The "error in percent" is

100 \times \frac{| θ_{t}^{n} - θ^{*, n} |}{| θ^{*, n} |}

Finally, we also track the objective function

\bar{g} (θ_{t})

over time. Figure 3 plots the error

\bar{g} (θ_{t})

against time. Since the limiting distribution

π (x)

of 5.2 is Gaussian with mean

m^{*}

and variance

\frac{1}{2 c^{*}}

, we have that:

\begin{aligned} \bar{g} (θ) & = \int {(c^{*} (m^{*} - x) - c (m - x))}^{2} π (x) d x \\ = {(c^{*} m^{*} - c m)}^{2} + {(c^{*} - c)}^{2} (\frac{1}{2 c^{*}} + {(m^{*})}^{2}) + 2 (c^{*} m^{*} - c m) (c - c^{*}) m^{*} \end{aligned}

Figure 3: The error

\bar{g} (θ_{t})

plotted against time. The mean error and the quantiles of the error are calculated from the 10,500 cases. Time is in log scale.

6.2. Multidimensional Ornstein-Uhlenbeck process

The multidimensional Ornstein-Uhlenbeck process

X_{t} \in R^{d}

satisfies the stochastic differential equation:

d X_{t} = (M - A X_{t}) d t + d W_{t} .

We use continuous-time stochastic gradient descent to learn the parameters

θ = (M, A) \in R^{d} \times R^{d \times d}

. For the numerical experiments, we use an Euler scheme with a time step of

10^{- 2}

. The learning rate is

α_{t} = min (α, α / t)

with

α = 10^{- 1}

. We simulate data from 5.2 for a particular

θ^{*} = (M^{*}, A^{*})

and the stochastic gradient descent attempts to learn a parameter

θ_{t}

which fits the data well.

θ_{t}

is the statistical estimate for

θ^{*}

at time

t

. If the estimation is accurate,

θ_{t}

should of course be close to

θ^{*}

. This example can be placed in the form of the original class of equations (1.1) by setting

f (x, θ) = M - A x

and

f^{*} (x) = f (x, θ^{*})

The matrix

A^{*}

must be generated carefully to ensure that

X_{t}

is ergodic and has a stable equilibrium point. If some of

A^{*}

's eigenvalues have negative real parts, then

X_{t}

can become unstable and grow arbitrarily large. Therefore, we randomly generate matrices

A^{*}

which are strictly diagonally dominant.

A^{*}

's eigenvalues are therefore guaranteed to have positive real parts and

X_{t}

will be ergodic. To generate random strictly diagonally dominant matrices

A^{*}

, we first generate

A_{i, j}^{*}

uniformly at random in the range [1,2] for

i \neq j

. Then, we set

A_{i, i}^{*} = \sum_{j \neq i} A_{i, j}^{*} + U_{i, i}

where

U_{i, i}

is generated randomly in

[1, 2] . M_{i}^{*}

for

i = 1, \dots, d

is also generated randomly in

[1, 2]

We study 525 cases and analyze the error in Table 2 Figures 4 and 5 plot the error over time.

Error/Time	$10^{2}$	$10^{3}$	$10^{4}$	$10^{5}$	$10^{6}$
Maximum Error	2.89	.559	.151	.043	.013
$99 %$ quantile of error	2.19	.370	.0957	.0294	.00911
$99.9 %$ quantile of error	2.57	.481	.118	.0377	.0117
Mean squared error	$8.05 \times 10^{- 1}$	$2.09 \times 10^{- 2}$	$1.38 \times 10^{- 3}$	$1.29 \times 10^{- 4}$	$1.25 \times 10^{- 5}$
Mean Error in percent	34.26	6.18	1.68	0.52	0.161
Maximum error in percent	186.3	41.68	10.98	3.81	1.03
$99 %$ quantile of error in percent	109.2	23.9	6.98	2.15	0.657
$99.9 %$ quantile of error in percent	141.2	31.24	8.64	2.84	0.879

Table 2: Error at different times for the estimate

θ_{t}

θ^{*}

across 525 cases. The "error" is

| θ_{t}^{n} - θ^{*, n} |

where

n

represents the

n

-th case. The "error in percent" is

100 \times \frac{| θ_{t}^{n} - θ^{*, n} |}{| θ^{*, n} |}

Figure 4: Mean error in percent plotted against time. Time is in log scale.

Figure 5: Mean squared error plotted against time. Time is in log scale.

6.3. Burger's Equation

The stochastic Burger's equation that we consider is given by:

\frac{\partial u}{\partial t} (t, x) = θ \frac{\partial^{2} u}{\partial x^{2}} - u (t, x) \frac{\partial u}{\partial x} (t, x) + σ \frac{\partial^{2} W (t, x)}{\partial t \partial x},

where

x \in [0, 1]

and

W (t, x)

is a Brownian sheet. The finite-difference discretization of

(5.3)

satisfies a system of nonlinear stochastic differential equations (for instance, see [13] or [3]). We use continuous-time stochastic gradient descent to learn the diffusion parameter

θ

We use the following finite difference scheme for Burger's equation:

d u (t, x_{i}) = θ \frac{u (t, x_{i + 1}) - 2 u (t, x_{i}) + u (t, x_{i - 1})}{Δ x^{2}} d t - u (t, x_{i}) \frac{u (t, x_{i + 1}) - u (t, x_{i - 1})}{2 Δ x} d t + \frac{σ}{\sqrt{Δ x}} d W_{t}^{i},

For our numerical experiment, the boundary conditions

u (t, x = 0) = 0

and

u (t, x = 1) = 1

are used and

σ = 0.1

. (5.4) is simulated with the Euler scheme (i.e., we solve Burger's equation with explicit finite difference). A spatial discretization of

Δ x = .01

and a time step of

10^{- 5}

are used. The learning rate is

α_{t} = min (α, α / t)

with

α = 10^{- 3}

. The small time step is needed to avoid instability in the explicit finite difference scheme. We simulate data from 5.3 for a particular diffusion coefficient

θ^{*}

and the stochastic gradient descent attempts to learn a diffusion parameter

θ_{t}

which fits the data well.

θ_{t}

is the statistical estimate for

θ^{*}

at time

t

. If the estimation is accurate,

θ_{t}

should of course be close to

θ^{*}

This example can be placed in the form of the original class of equations (1.1). Let

f_{i}

be the

i

-th element of the function

f

. Then,

f_{i} (u, θ) = θ \frac{u (t, x_{i + 1}) - 2 u (t, x_{i}) + u (t, x_{i - 1})}{Δ x^{2}} - u (t, x_{i}) \frac{u (t, x_{i + 1}) - u (t, x_{i - 1})}{2 Δ x}

. Similarly, let

f_{i}^{*}

be the

i

-th element of the function

f^{*}

. Then,

f_{i}^{*} (u) = f_{i} (u, θ^{*})

We study 525 cases. For each case, a different

θ^{*}

is generated uniformly at random in the range [.1,10]. This represents a wide range of physical cases of interest, with

θ^{*}

ranging over two orders of magnitude. For each case, we solve for the parameter

θ_{t}

over the time period

[0, T]

for

T = 100

The accuracy of

θ_{t}

at times

t = 10^{- 1}, 10^{0}, 10^{1}

, and

10^{2}

is reported in Table 3 . Figures 6 and 7 plot the mean error in percent and mean squared error against time. The convergence of

θ_{t}

θ^{*}

is fairly rapid in time.

Error/Time	$10^{- 1}$	$10^{0}$	$10^{1}$	$10^{2}$
Maximum Error	.1047	.106	.033	.0107
99% quantile of error	.08	.078	.0255	.00835
Mean squared error	$1.00 \times 10^{- 3}$	$9.25 \times 10^{- 4}$	$1.02 \times 10^{- 4}$	$1.12 \times 10^{- 5}$
Mean Error in percent	1.26	1.17	0.4	0.13
Maximum error in percent	37.1	37.5	9.82	4.73
99% quantile of error in percent	12.6	18.0	5.64	1.38

Table 3: Error at different times for the estimate

θ_{t}

θ^{*}

across 525 cases. The "error" is

| θ_{t}^{n} - θ^{*, n} |

where

n

represents the

n

-th case. The "error in percent" is

100 \times | θ_{t}^{n} - θ^{*, n} | / | θ^{*, n} |

Figure 6: Mean error in percent plotted against time. Time is in log scale.

6.4. Reinforcement Learning

We consider the classic reinforcement learning problem of balancing a pole on a moving cart (see [6]). The goal is to balance a pole on a cart and to keep the cart from moving outside the boundaries via applying a force of \pm 10 Newtons.

The position

x

of the cart, the velocity

\dot{x}

of the cart, angle of the pole

β

, and angular velocity

\dot{β}

of the pole are observed. The dynamics of

s = (x, \dot{x}, β, \dot{β})

satisfy a set of ODEs (see [6]):

\begin{aligned} {\ddot{β}}_{t} = \frac{g \sin β_{t} + \cos β_{t} [\frac{- F_{t} - m l {\dot{β}}_{t}^{2} \sin β_{t} + μ_{c} sgn ({\dot{x}}_{t})}{m_{c} + m}] - \frac{μ_{p} {\dot{β}}_{t}}{m l}}{l [\frac{4}{3} - \frac{m}{m_{c}} \frac{\cos^{2} β_{t}}{m_{c} + m}]}, \\ {\ddot{x}}_{t} = \frac{F_{t} + m l [{\dot{β}}_{t}^{2} \sin β_{t} - {\ddot{β}}_{t} \cos β_{t}] - μ_{c} sgn ({\dot{x}}_{t})}{m_{c} + m}, \end{aligned}

where

g

is the acceleration due to gravity,

m_{c}

is the mass of the cart,

m

is the mass of the pole,

2 l

is the length of the pole,

μ_{c}

is the coefficient of friction of the cart on the ground,

μ_{p}

is the coefficient of friction of the pole on the cart, and

F_{t} \in {- 10, 10}

is the force applied to the cart.

For this example,

f^{*} (s) = (\dot{x}, \ddot{x}, \dot{β}, \ddot{β})

. The model

f (s, θ) = (f_{1} (s, θ), f_{2} (s, θ), f_{3} (s, θ), f_{4} (s, θ))

where

f_{i} (s, θ)

is a single-layer neural network with rectified linear units.

f_{i} (s, θ) = W^{2, i} h (W^{1, i} s + b^{1, i}) + b^{2, i},

Figure 7: Mean squared error plotted against time. Time is in log scale.

where

θ = {W^{2, i}, W^{1, i}, b^{1, i}, b^{2, i}}_{i = 1}^{4}

and

h (z) = (σ (z_{1}), \dots, σ (z_{d}))

for

z \in R^{d}

. The function

σ : R \to R

is a rectified linear unit (ReLU):

σ (v) = max (v, 0)

. We learn the parameter

θ

using continuous-time stochastic gradient descent.

The boundary is

x = \pm 2.4

meters and the pole must not be allowed to fall below

β = \frac{24}{360 π}

radians (the frame of reference is chosen such that the perfectly upright is 0 radians). A reward of +1 is received every 0.02 seconds if

∥ x ∥ \leq 2.4

and

∥ θ ∥ \leq \frac{24}{360 π}

. A reward of -100 is received (and the episode ends) if the cart moves beyond

x = \pm 2.4

or the pole falls below

β = \frac{24}{360 π}

radians. The sum of these rewards across the entire episode is the reward for that episode. The initial state

(x, \dot{x}, β, \dot{β})

at the start of an episode is generated uniformly at random in

[- .05, .05]^{4}

. For our numerical experiment, we assume that the rule for receiving the rewards and the distribution of the initial state are both known. An action of \pm 10 Newtons may be chosen every 0.02 seconds. This force is then applied for the duration of the next 0.02 seconds. The system (5.5) is simulated using an Euler scheme with a time step size of

10^{- 3}

seconds.

The goal, of course, is to statistically learn the optimal actions in order to achieve the highest possible reward. This requires both: 1) statistically learning the physical dynamics of

(x, \dot{x}, β, \dot{β})

and 2

)

finding the optimal actions given these dynamics in order to achieve the highest possible reward. The dynamics

(x, \dot{x}, β, \dot{β})

satisfy the set of ODEs

(5.5)

; these dynamics can be learned using continuous-time stochastic gradient descent. We use a neural network for

f

. Given the estimated dynamics

f

, we use a policy gradient method to estimate the optimal actions. The approach is summarized below.

For episodes $0, 1, 2, \dots$ :
For time $[0, T_{end of episode}]$ :

Update the model $f (s, θ)$ for the dynamics using continuous-time stochastic gradient descent.

Periodically update the optimal policy $μ (s, a, θ^{μ})$ using policy gradient method. The optimal policy is learned using data simulated from the model $f (s, θ)$ . Actions are randomly selected via the policy $μ$ .

The policy

μ

is a neural network with parameters

θ^{μ}

. We use a single hidden layer with rectified linear units followed by a softmax layer for

μ (s, a, θ^{μ})

and train it using policy gradients 1 The policy

μ (s, a, θ^{μ})

gives the probability of taking action

a

conditional on being in the state

s

P [F_{t} = 10 ∣ s_{t} = s] = μ (s_{t}, 10, θ^{μ}) = σ_{0} (W^{2} h (W^{1} s + b^{1}) + b^{2}),

where

σ_{0} (v) = \frac{e^{v}}{1 + e^{v}}

. Of course,

P [F_{t} = - 10 ∣ s_{t} = s] = μ (s, - 10, θ^{μ}) = 1 - μ (s, 10, θ^{μ})

525 cases are run, each for 25 hours. The optimal policy is learned using the estimated dynamics

f (s, θ)

and is updated every 5 episodes. Table 4 reports the results at fixed episodes using continuous-time stochastic gradient descent. Table 5 reports statistics on the number of episodes required until a target episodic reward

(100, 500, 1000)

is first achieved.

Reward/Episode	10	20	30	40	45
Maximum Reward	-20	981	$2.21 \times 10^{4}$	$6.64 \times 10^{5}$	$9.22 \times 10^{5}$
90% quantile of reward	-63	184	760	8354	$1.5 \times 10^{4}$
Mean reward	-78	67	401	5659	$1.22 \times 10^{4}$
10% quantile of reward	-89	-34	36	69	93
Minimum reward	-92	-82	-61	-46	-23

Table 4: Reward at the

k

-th episode across the 525 cases using continuous-time stochastic gradient descent to learn the model dynamics.

Number of episodes/Target reward	100	500	1000
Maximum	39	134	428
90% quantile	23	49	61
Mean	18	34	43
10% quantile	13	21	26
Minimum	11	14	17

Table 5: For each case, we record the number of episodes required until the target reward is first achieved using continuous-time stochastic gradient descent. Statistics (maximum, quantiles, mean, minimum) for the number of episodes required until the target reward is first achieved.

Alternatively, one could directly apply policy gradient to learn the optimal action using the observed data. This approach does not use continuous-time stochastic gradient descent to learn the model dynamics, but instead directly learns the optimal policy from the data. Again using 525 cases, we report the results in Table 6 for directly learning the optimal policy without using continuous-time stochastic gradient descent to learn the model dynamics. Comparing Tables 4 and 6 , it is clear that using continuous-time stochastic gradient descent to learn the model dynamics allows for the optimal policy to be learned significantly more quickly. The rewards are much higher when using continuous-time stochastic gradient descent (see Table 4 ) than when not using it (see Table6).

Reward/Episode	10	20	30	40	100	500	750
Maximum Reward	51	1	15	77	121	1748	$1.91 \times 10^{5}$
90% quantile of reward	-52	-48	-42	8354	-11	345	2314
Mean reward	-73	-72	-69	-68	-53	150	1476
10% quantile of reward	-88	-88	-87	69	-83	-1	63
Minimum reward	-92	-92	-92	-92	-92	-81	-74

Table 6: Reward at the

k

-th episode across the 525 cases using policy gradient to learn the optimal policy.

6.5. Estimating both the drift and volatility functions for the multidimensional CIR process

We now implement an example where SGDCT is used to estimate both the drift function and the volatility function. The multidimensional CIR process

X_{t} \in R^{d}

is:

d X_{t} = c (m - X_{t}) d t + \sqrt{X_{t}} ⊙ σ d W_{t},

where

⊙

is element-wise multiplication,

m \in R^{d}, c, σ \in R^{d \times d}, W_{t} \in R^{d}

, with

c

being a positive definite matrix. The CIR process is often used for modeling interest rates.

In equation (5.8),

f (x, θ) = c (m - x)

where

θ = (c, m) . f^{*} (x) = f (x, θ^{*})

where

θ^{*} = (c^{*}, m^{*})

. The volatility model is

σ (x, ν) = \sqrt{x} ⊙ ν

and

σ^{*} (x) = σ (x, ν^{*})

where

ν, ν^{*} \in R^{d \times d}

. Table 7 reports the accuracy of SGDCT for estimating the drift and volatility functions of the CIR process.

Error/Parameter	$c$	$m$	${(\sqrt{X_{t}} ⊙ σ)}^{⊤} (\sqrt{X_{t}} ⊙ σ)$
Maximum Error	0.0157	0.009	0.010
99% quantile of error	0.010	0.007	0.008
$99.9 %$ quantile of error	0.0146	0.009	0.010
Mean squared error	$1.49 \times 10^{- 5}$	$6.65 \times 10^{- 6}$	$4.21 \times 10^{- 6}$
Mean Error in percent	0.21	0.137	0.0623
Maximum error in percent	1.12	0.695	0.456
$99 %$ quantile of error in percent	0.782	0.506	0.415
$99.9 %$ quantile of error in percent	1.06	0.616	0.455

Table 7: Accuracy is reported in percent and averaged across 317 simulations. Each simulation has a different random initialization for

c, m

, and

σ

. The dimension

d = 3

, the time step size is

10^{- 2}

, and accuracy is evaluated at the final time

5 \times 10^{5}

X_{t}

is simulated using 4. Observations of the quadratic variation are generated from

{(\sqrt{X_{t}} ⊙ σ)}^{⊤} (\sqrt{X_{t}} ⊙ σ)

at times

t = 0, .01, .02, \dots {(\sqrt{X_{t}} ⊙ σ)}^{⊤} (\sqrt{X_{t}} ⊙ σ)

is the quadratic variation per unit of time. For each simulation, the average error (or average percent error) for the quadratic variation per unit time is calculated by averaging across many points in the path of

X_{t}

. Then, the statistics in the third column of the table are calculated using the average errors (or average percent errors) from the 317 simulations.

7. American Options

High-dimensional American options are extremely computationally challenging to solve with traditional numerical methods such as finite difference. Here we propose a new approach using statistical learning to solve high-dimensional American options. SGDCT achieves a high accuracy on two benchmark problems with 100 dimensions.

7.1. Q-learning

Before describing the SGDCT algorithm for American options, it is important to note that traditional stochastic gradient descent faces certain difficulties in this class of problems. Some brief remarks are provided below regarding this fact; the authors plan to elaborate on these issues in more detail in a future work. The well-known Q-learning algorithm uses stochastic gradient descent to minimize an approximation to the discrete-time Hamilton-Jacobi-Bellman equation. To demonstrate the challenges and the issues that arise, consider using Q-learning to estimate the value function:

V (x) = E [\int_{0}^{\infty} e^{- γ t} r (X_{t}) d t ∣ X_{0} = x], X_{t} = x + W_{t},

where

γ > 0

is a discount factor and

r (x)

is a reward function. The function

Q (x, θ)

is an approximation for the value function

V (x)

. The parameter

θ

must be estimated. The traditional approach would discretize the dynamics (6.1) and then apply a stochastic gradient descent update to the objective function:

E [{(r (X_{t}) Δ + E [e^{- γ Δ} Q (X_{t + Δ}; θ) ∣ X_{t}] - Q (X_{t} θ))}^{2}] .

This results in the stochastic gradient descent algorithm:

\begin{aligned} θ_{t + Δ} & = θ_{t} - \frac{α_{t}}{Δ} (e^{- γ Δ} E [Q_{θ} (X_{t + Δ}; θ_{t}) ∣ X_{t}] - Q_{θ} (X_{t}; θ_{t})) \\ \times (r (X_{t}) Δ + e^{- γ Δ} E [Q (X_{t + Δ}; θ_{t}) ∣ X_{t}] - Q (X_{t}; θ_{t})) . \end{aligned}

Note that we have scaled the learning rate in 6.3 by

\frac{1}{Δ}

. This is the correct scaling for taking the limit

Δ \to 0

. The algorithm (6.3) has a major computational issue. If the process

X_{t}

is high-dimensional,

E [Q (X_{t + Δ}; θ_{t}) ∣ X_{t}]

is computationally challenging to calculate, and this calculation must be repeated for a large number of samples (millions to hundreds of millions). It is also important to note that for the American option example that follows the underlying dynamics are known. However, in reinforcement learning applications the transition probability is unknown, in which case

E [Q (X_{t + Δ}; θ_{t}) ∣ X_{t}]

cannot be calculated. To circumvent these obstacles, the Q-learning algorithm ignores the inner expectation in 6.2 , leading to the algorithm:

θ_{t + Δ} = θ_{t} - \frac{α_{t}}{Δ} (e^{- γ Δ} Q_{θ} (X_{t + Δ}; θ_{t}) - Q_{θ} (X_{t}; θ_{t})) (r (X_{t}) Δ + e^{- γ Δ} Q (X_{t + Δ}; θ_{t}) - Q (X_{t}; θ_{t}))

Although now computationally efficient, the Q-learning algorithm 6.4 is now biased (due to ignoring the inner expectations). Furthermore, when

Δ \to 0

, the Q-learning algorithm (6.4) blows up. A quick investigation shows that the term

\frac{1}{Δ} {(W_{t + Δ} - W_{t})}^{2} = O (1)

arises while all other terms are

O (Δ)

O (\sqrt{Δ})

The SGDCT algorithm is unbiased and computationally efficient. It can be directly derived by letting

Δ \to 0

and using Itô's formula in

(6.3)

d θ_{t} = - α_{t} (\frac{1}{2} Q_{θ x x} (X_{t}; θ_{t}) - γ Q_{θ} (X_{t}; θ_{t})) (r (X_{t}) + \frac{1}{2} Q_{x x} (X_{t}; θ_{t}) - γ Q (X_{t}; θ_{t})) d t .

Note that computationally challenging terms in (6.3) become differential operators in (6.7), which are usually easier to evaluate. This is one of the advantages of developing the theory in continuous time for continuoustime models. Once the continuous-time algorithm is derived, it can be appropriately discretized for numerical solution.

7.2. SGDCT for American Options

Let

X_{t} \in R^{d}

be the prices of

d

stocks. The maturity date is time

T

and the payoff function is

g (x) : R^{d} \to R

. The stock dynamics and value function are:

\begin{aligned} d X_{t}^{i} & = μ (X_{t}^{i}) d t + σ (X_{t}^{i}) d W_{t}^{i}, \\ V (t, x) & = sup_{τ \geq t} E [e^{- r (τ \land T)} g (X_{τ \land T}) ∣ X_{t} = x], \end{aligned}

where

W_{t} \in R^{d}

is a Brownian motion. The distribution of

W_{t}

is specified by

Var [W_{t}^{i}] = t

and

Corr [W_{t}^{i}, W_{t}^{j}] =

ρ_{i, j}

for

i \neq j

. The SGDCT algorithm for an American option is:

\begin{aligned} θ_{τ \land T}^{n + 1} & = θ_{0}^{n} - \int_{0}^{τ \land T} α_{t}^{n + 1} (\frac{\partial}{\partial t} Q_{θ} (t, X_{t}; θ_{t}^{n + 1}) + L_{x} Q_{θ} (t, X_{t}; θ_{t}^{n + 1}) - r Q_{θ} (t, X_{t}; θ_{t}^{n + 1})) \\ \times (\frac{\partial Q}{\partial t} (t, X_{t}; θ_{t}^{n + 1}) + L_{x} Q (t, X_{t}; θ_{t}^{n + 1}) - r Q (t, X_{t}; θ_{t}^{n + 1})) d t \\ + α_{τ \land T}^{n + 1} Q_{θ} (τ \land T, X_{τ \land T}; θ_{τ \land T}^{n + 1}) (g (X_{τ \land T}) - Q (τ \land T, X_{τ \land T}; θ_{τ \land T}^{n + 1})), \\ τ & = inf {t \geq 0 : Q (t, X_{t}; θ_{t}^{n + 1}) < g (X_{t})} \\ X_{0} & \sim ν (d x) . \end{aligned}

L_{x}

is the infinitesimal generator for the

X

process. The continuous-time algorithm 6.7) is run for many iterations

n = 0, 1, 2, \dots

until convergence. See the authors' paper 28] for implementation details on pricing American options with deep learning.

We implement the SGDCT algorithm (6.7) using a deep neural network for the function

Q (t, x; θ)

. Two benchmark problems are considered where semi-analytic solutions are available. The SGDCT algorithm's accuracy is evaluated for American options in

d = 100

dimensions, and the results are presented in Table 8

Model	Number of dimensions	Payoff function	Accuracy
Bachelier	100	$g (x) = max (\frac{1}{d} \sum_{i = 1}^{d} x_{i} - K, 0)$	$0.1 %$
Black-Scholes	100	$g (x) = max ({(\prod_{i = 1}^{d} x_{i})}^{1 / d} - K, 0)$	$0.2 %$

Table 8: For the Bachelier model,

μ (x) = r - c

and

σ (x) = σ

. For Black-Scholes,

μ (x) = (r - c) x

and

σ (x) = σ x

. All stocks are identical with correlation

ρ_{i, j} = .75

, volatility

σ = .25

, initial stock price

X_{0} = 1

, dividend rate

c = 0.02

, and interest rate

r = 0

. The maturity of the option is

T = 2

and the strike price is

K = 1

. The accuracy is reported for the price of the at-the-money American call option.

We recall the following regularity result from 24] on the Poisson equations in the whole space, appropriately stated to cover our case of interest.

Theorem A.1. Let Conditions 2.2 and 2.3 be satisfied. Assume that

G (x, θ) \in C^{α, 2} (X, R^{n})

\int_{X} G (x, θ) π (d x) = 0

and that for some positive constants

K

and

q

\sum_{i = 0}^{2} | \frac{\partial^{i} G}{\partial θ^{i}} (x, θ) | \leq K (1 + | x |^{q})

Let

L_{x}

be the infinitesimal generator for the

X

process. Then the Poisson equation

L_{x} u (x, θ) = G (x, θ), \int_{X} u (x, θ) π (d x) = 0

has a unique solution that satisfies

u (x, \cdot) \in C^{2}

for every

x \in X, \partial_{θ}^{2} u \in C (X \times R^{n})

and there exist positive constants

K^{'}

and

q^{'}

such that

\sum_{i = 0}^{2} | \frac{\partial^{i} u}{\partial θ^{i}} (x, θ) | + | \frac{\partial^{2} u}{\partial x \partial θ} (x, θ) | \leq K^{'} (1 + | x |^{q^{'}}) .

9. References

[1] Y. Ait-Sahalia, Maximum Likelihood Estimation of Discretely Sampled Diffusions: A Closed-form Approximation Approach, Econometrica, Vol. 70, No. 1, 2002, pp. 223-262.

[2] Y. Ait-Sahalia, Closed-form likelihood expansions for multivariate diffusions, Annals of Statistics, Vol. 36, No. 2, 2008, pp. 906-937.

[3] A. Alabert and I. Gyongy, On numerical approximation of stochastic Burger's equation, From stochastic calculus to mathematical finance, Springer Berling Heidelberg, 2006. 1-15.

[4] A. Alfonsi, High order discretization schemes for the CIR process: application to affine term structure and Heston models, Mathematics of Computation, Vol. 79, No. 269, 2010, pp. 209-237.

[5] R. Ahlip and M. Rutkowski, Pricing of foreign exchange options under the Heston stochastic volatility model and CIR interest rates, Quantitative Finance, Vol. 13, No. 6, 2013, pp. 955-966.

[6] A. Barto, R. Sutton, and C. Anderson, Neuronlike Adaptive Elements that can solve difficult learning control problem, IEEE Transactions on Systems, Man, and Cybernetics, Vol.5, 1983, pp. 834-846.

[7] I. Basawa and B. Rao, Asymptotic inference for stochastic processes, Stochastic Processes and their Applications, Vol.10, No. 3, 1980, pp. 221-254.

[8] A. Benveniste, M. Metivier, and P. Priouret, Adaptive Algorithms and Stochastic Approximations. Springer-Verlag, 2012.

[9] Dimitri P. Bertsekas and John N. Tsitsiklis, Gradient convergence in gradient methods via errors, SIAM Journal of Optimization, Vol.10, No. 3, 2000, pp. 627-642.

[10] J. P. N. Bishwal, Parameter estimation in stochastic differential equations, in: Lecture Notes in Mathematics, Vol. 1923, Springer Science & Business Media, 2008.

[11] S. Brown and P. Dybvig, The empirical implications of the Cox, Ingersoll, Ross theory of the term structure of interest rates, The Journal of Finance, Vol.41, No. 3, 1986, pp. 617-630.

[12] Q. Dai and K. Singleton, Specification analysis of affine term structure models, The Journal of Finance, Vol. 55, No. 5, 2000, pp. 1943-1978.

[13] A. Davie and J. Gaines, Convergence of numerical schemes for the solution of parabolic stochastic partial differential equations, Mathematics of Computation, Vol. 70, No. 233, (2001), pp. 121-134.

[14] O. Elerian, S. Chib, and N. Shephard, Likelihood inference for discretely observed nonlinear diffusions, Econometrica, Vol. 69, No. 4, (2001), pp. 959-993.

[15] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Book in preparation for MIT Press, 2016.

[16] H. Kushner and G. Yin, Stochastic Approximation and Recursive Algorithms and Applications, Second Edition. Springer, 2003.

[17] Y. Kutoyants, Statistical inference for ergodic diffusion processes, Springer Science & Business Media, 2004.

[18] V. Linetsky, Computing hitting time densities for CIR and OU diffusions: applications to mean-reverting models, Journal of Computational Finance, Vol. 7, 2004, pg. 1-22. [19] T. Leung and X. Li, Optimal mean reversion trading with transaction costs and stop-loss exit, International Journal of Theoretical and Applied Finance, Vol. 18, No. 3, 2015, pg. 155020.

[20] T. Leung, J. Li, X. Li, and Z. Wang, Speculative futures trading under mean reversion, Asia-Pacific Financial Markets, Vol. 23, No. 4, 2015, pp. 281-304.

[21] A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, Robust stochastic approximation to stochastic programming, SIAM Journal of Optimization, Vol.19, No. 4, 2009, pp. 1574-1609.

[22] Y. Maghsoodi, Solution of the extended CIR term structure and bond option valuation, Mathematical Finance, Vol. 6, No. 1, 1996, pp. 89-109.

[23] E. Pardoux and A.Yu. Veretennikov, On Poisson equation and diffusion approximation 1, Annals of Probability, Vol. 29, No. 3, 2001, pp. 1061-1085.

[24] E. Pardoux and A. Y. Veretennikov, On Poisson equation and diffusion approximation 2, The Annals of Probability, Vol. 31, No. 3, 2003, pp. 1166-1192.

[25] M. Raginsky and J. Bouvrie, Continuous-time stochastic mirror descent on a network: variance reduction, consensus, convergence, IEEE Conference on Decision and Control, 2012.

[26] B. L. S. P. Rao, Statistical inference for diffusion type processes, Arnold, 1999.

[27] G. O. Roberts and R.L. Tweedie, Exponential convergence of Langevin distributions and their discrete approximations, Bernoulli, 1996, pp. 341-363.

[28] J. Sirignano and K. Spiliopoulos, DGM: A deep learning algorithm for solving partial differential equations, 2017, arXiv: 1708.07469.

[29] O. Vasicek, An equilibrium characterization of the term structure, Journal of financial economics, Vol. 5, No. 2, 1977, pp. 177-188.

*Department of Industrial and Enterprise Systems Engineering, University of Illinois at Urbana-Champaign, Urbana, Email: jasirign@illinois.edu

$^{†}$ Department of Mathematics and Statistics, Boston University, Boston, E-mail: kspiliop@math.bu.edu

$^{‡}$ Research of K.S. supported in part by the National Science Foundation (DMS 1550918)

$§$ Computations for this paper were supported by a Blue Waters supercomputer grant.
$^{1}$ Let $r_{e, t}$ be the reward for episode $e$ at time $t$ . Let $R_{t, e} = \sum_{t^{'} = t + 1}^{T_{end} of episode} γ^{t^{'} - t} r_{e, t^{'}}$ be the cumulative discounted reward from episode $e$ after time $t$ where $γ \in [0, 1]$ is the discount factor. Stochastic gradient descent is used to learn the parameter $θ^{μ}$ : $θ^{μ} \leftarrow θ^{μ} + η_{e} R_{t, e} \frac{\partial}{\partial θ^{μ}} \log μ (s_{t}, a_{t}, θ^{μ})$ where $η_{e}$ is the learning rate. In practice, the cumulative discounted rewards are often normalized across an episode.

Stochastic Gradient Descent in Continuous Time

Abstract

1. Introduction

2. 1.1 1.1 1.1quad1.1 \quad1.1 Literature Review

2.1. Applications of SGDCT

2.2. Organization of Paper

3. Assumptions and Main Result

4. Proof of Theorem 2.4

5. Estimating the Coefficient Function of the Diffusion Term and Generalizations

6. Model Estimation: Numerical Analysis

6.1. Ornstein-Uhlenbeck process

6.2. Multidimensional Ornstein-Uhlenbeck process

6.3. Burger's Equation

6.4. Reinforcement Learning

6.5. Estimating both the drift and volatility functions for the multidimensional CIR process

7. American Options

7.1. Q-learning

7.2. SGDCT for American Options

8. A On a related Poisson equation

9. References

2. $1.1$ Literature Review